{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fsv4jGuU89rX"
   },
   "source": [
    "# Monitoring Vertex AI Model \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lA32H1oKGgpf"
   },
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t6Cd51FkG09E"
   },
   "source": [
    "### What is Model Monitoring?\n",
    "\n",
    "Modern applications rely on a well established set of capabilities to monitor the health of their services. Examples include:\n",
    "\n",
    "* software versioning\n",
    "* rigorous deployment processes\n",
    "* event logging\n",
    "* alerting/notification of situations requiring intervention\n",
    "* on-demand and automated diagnostic tracing\n",
    "* automated performance and functional testing\n",
    "\n",
    "You should be able to manage your ML services with the same degree of power and flexibility with which you can manage your applications. That's what MLOps is all about - managing ML services with the best practices Google and the broader computing industry have learned from generations of experience deploying well engineered, reliable, and scalable services.\n",
    "\n",
    "Model monitoring is only one piece of the ML Ops puzzle - it helps answer the following questions:\n",
    "\n",
    "* How well do recent service requests match the training data used to build your model? This is called **training-serving skew**.\n",
    "* How significantly are service requests evolving over time? This is called **drift detection**.\n",
    "\n",
    "If production traffic differs from  training data, or varies substantially over time, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yG7FcXWKHOhC"
   },
   "source": [
    "## Learning Objectives\n",
    "\n",
    "1. Deploy a pre-trained model.\n",
    "2. Configure model monitoring.\n",
    "3. Generate some artificial traffic.\n",
    "4. Interpret the data reported by the model monitoring feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8yVpQt-JHKPF"
   },
   "source": [
    "## Introduction \n",
    "\n",
    "In this notebook, you will deploy a pre-trained model to an endpoint and generate some prediction requests on the model.\n",
    "You will also create a monitoring job to keep an eye on the model quality and generate test data to trigger alerting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "### The example model\n",
    "\n",
    "The model you'll use in this notebook is based on [this blog post](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:\n",
    "\n",
    "- identity - unique player identitity numbers\n",
    "- demographic features - information about the player, such as the geographic region in which a player is located\n",
    "- behavioral features - counts of the number of times a  player has triggered certain game events, such as reaching a new level\n",
    "- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player.\n",
    "\n",
    "The blog article referenced above explains how to use BigQuery to store the raw data, pre-process it for use in machine learning, and train a model. Because this notebook focuses on model monitoring, rather than training models, you're going to reuse a pre-trained version of this model, which has been exported to Google Cloud Storage. In the next section, you will setup your environment and import this model into your own project.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/model_monitoring.ipynb) -- try to complete that notebook first before reviewing this solution notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ze4-nDLfK4pw"
   },
   "source": [
    "## Before you begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Setup your dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "020040f91150"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "UcwUlnMnfo3i"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing TensorFlow 2.4.1 and TensorFlow Data Validation (TFDV)\n",
      "\u001b[33m  WARNING: The script plasma_store is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[33m  WARNING: The scripts estimator_ckpt_converter, import_pb_to_tensorboard, saved_model_cli, tensorboard, tf_upgrade_v2, tflite_convert, toco and toco_from_protos are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow-io 0.18.0 requires tensorflow-io-gcs-filesystem==0.18.0, which is not installed.\n",
      "tensorflow-transform 1.2.0 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<2.6,>=1.15.2, but you have tensorflow 2.4.1 which is incompatible.\n",
      "tensorflow-transform 1.2.0 requires tensorflow-metadata<1.3.0,>=1.2.0, but you have tensorflow-metadata 0.30.0 which is incompatible.\n",
      "tensorflow-transform 1.2.0 requires tfx-bsl<1.3.0,>=1.2.0, but you have tfx-bsl 0.30.0 which is incompatible.\n",
      "tensorflow-io 0.18.0 requires tensorflow<2.6.0,>=2.5.0, but you have tensorflow 2.4.1 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.8 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tfx-bsl 0.30.0 requires google-api-python-client<2,>=1.7.11, but you have google-api-python-client 2.18.0 which is incompatible.\n",
      "tensorflow-transform 1.2.0 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<2.6,>=1.15.2, but you have tensorflow 2.4.1 which is incompatible.\n",
      "tensorflow-transform 1.2.0 requires tensorflow-metadata<1.3.0,>=1.2.0, but you have tensorflow-metadata 0.30.0 which is incompatible.\n",
      "tensorflow-transform 1.2.0 requires tfx-bsl<1.3.0,>=1.2.0, but you have tfx-bsl 0.30.0 which is incompatible.\n",
      "jupyterlab-git 0.11.0 requires nbdime<2.0.0,>=1.1.0, but you have nbdime 3.1.0 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 2.18.0 which is incompatible.\u001b[0m\n",
      "\u001b[33m  WARNING: The script tb-gcp-uploader is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "Restarting kernel...\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "# Import necessary libraries\n",
    "import os\n",
    "import sys\n",
    "\n",
    "import IPython\n",
    "\n",
    "assert sys.version_info.major == 3, \"This notebook requires Python 3.\"\n",
    "\n",
    "# Install Python package dependencies.\n",
    "# Upgrade the specified packages to the available versions\n",
    "print(\"Installing TensorFlow Data Validation (TFDV)\")\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade tensorflow_data_validation[visualization]\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade google-api-python-client google-auth-oauthlib google-auth-httplib2 oauth2client requests\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-aiplatform\n",
    "! pip3 install {USER_FLAG} --quiet --upgrade google-cloud-storage\n",
    "\n",
    "# Automatically restart kernel after installing new packages.\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    print(\"Restarting kernel...\")\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)\n",
    "    print(\"Done.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "8rv5wnmA6BTd"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import sys\n",
    "import time\n",
    "\n",
    "# Import required packages.\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BF1j6f9HApxa"
   },
   "source": [
    "### Set up your Google Cloud project\n",
    "\n",
    "**For this lab, we're going to use region us-central1 for all our resources (BigQuery training data, Cloud Storage bucket, model and endpoint locations, etc.). Those resources can be deployed in other regions, as long as they're consistently co-located, but we're going to use one fixed region to keep things as simple and error free as possible.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cellView": "form",
    "id": "wxiE6dEWOFm3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n",
      "Updated property [ai/region].\n"
     ]
    }
   ],
   "source": [
    "# Make sure to replace [your-project-id] with your GCP project ID\n",
    "PROJECT_ID = \"[your-project-id]\"\n",
    "REGION = \"us-central1\"\n",
    "SUFFIX = \"aiplatform.googleapis.com\"\n",
    "API_ENDPOINT = f\"{REGION}-{SUFFIX}\"\n",
    "PREDICT_API_ENDPOINT = f\"{REGION}-prediction-{SUFFIX}\"\n",
    "if os.getenv(\"IS_TESTING\"):\n",
    "    !gcloud --quiet components install beta\n",
    "    !gcloud --quiet components update\n",
    "!gcloud config set project $PROJECT_ID\n",
    "!gcloud config set ai/region $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sECqTau7Oh6M"
   },
   "source": [
    "### Login to your Google Cloud account and enable AI services"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "C6H1vZYjvT6w"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Operation \"operations/acf.p2-346810986095-756e3ab7-4ed8-430a-b9b7-7a7a26e4023b\" finished successfully.\n"
     ]
    }
   ],
   "source": [
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# If on Google Cloud Notebooks, then don't execute this code\n",
    "if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    if \"google.colab\" in sys.modules:\n",
    "        from google.colab import auth as google_auth\n",
    "\n",
    "        google_auth.authenticate_user()\n",
    "\n",
    "    # If you are running this notebook locally, replace the string below with the\n",
    "    # path to your service account key and run this cell to authenticate your GCP\n",
    "    # account.\n",
    "    elif not os.getenv(\"IS_TESTING\"):\n",
    "        %env GOOGLE_APPLICATION_CREDENTIALS ''\n",
    "\n",
    "# Enable AI services\n",
    "!gcloud services enable aiplatform.googleapis.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "btZeLzqQ7pXc"
   },
   "source": [
    "### Define some helper functions\n",
    "\n",
    "Run the following cell to define some utility functions used throughout this notebook. Although these functions are not critical to understand the main concepts, feel free to expand the  cell if you're curious or want to dive deeper into how some of your API requests are made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "yhDFSB2YDvfT"
   },
   "outputs": [],
   "source": [
    "# @title Utility functions\n",
    "import copy\n",
    "import os\n",
    "\n",
    "from google.cloud.aiplatform_v1beta1.services.endpoint_service import \\\n",
    "    EndpointServiceClient\n",
    "from google.cloud.aiplatform_v1beta1.services.job_service import \\\n",
    "    JobServiceClient\n",
    "from google.cloud.aiplatform_v1beta1.services.prediction_service import \\\n",
    "    PredictionServiceClient\n",
    "from google.cloud.aiplatform_v1beta1.types.io import BigQuerySource\n",
    "from google.cloud.aiplatform_v1beta1.types.model_deployment_monitoring_job import (\n",
    "    ModelDeploymentMonitoringJob, ModelDeploymentMonitoringObjectiveConfig,\n",
    "    ModelDeploymentMonitoringScheduleConfig)\n",
    "from google.cloud.aiplatform_v1beta1.types.model_monitoring import (\n",
    "    ModelMonitoringAlertConfig, ModelMonitoringObjectiveConfig,\n",
    "    SamplingStrategy, ThresholdConfig)\n",
    "from google.cloud.aiplatform_v1beta1.types.prediction_service import \\\n",
    "    PredictRequest\n",
    "from google.protobuf import json_format\n",
    "from google.protobuf.duration_pb2 import Duration\n",
    "from google.protobuf.struct_pb2 import Value\n",
    "\n",
    "DEFAULT_THRESHOLD_VALUE = 0.001\n",
    "\n",
    "\n",
    "def create_monitoring_job(objective_configs):\n",
    "    # Create sampling configuration.\n",
    "    random_sampling = SamplingStrategy.RandomSampleConfig(sample_rate=LOG_SAMPLE_RATE)\n",
    "    sampling_config = SamplingStrategy(random_sample_config=random_sampling)\n",
    "\n",
    "    # Create schedule configuration.\n",
    "    duration = Duration(seconds=MONITOR_INTERVAL)\n",
    "    schedule_config = ModelDeploymentMonitoringScheduleConfig(monitor_interval=duration)\n",
    "\n",
    "    # Create alerting configuration.\n",
    "    emails = [USER_EMAIL]\n",
    "    email_config = ModelMonitoringAlertConfig.EmailAlertConfig(user_emails=emails)\n",
    "    alerting_config = ModelMonitoringAlertConfig(email_alert_config=email_config)\n",
    "\n",
    "    # Create the monitoring job.\n",
    "    endpoint = f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}\"\n",
    "    predict_schema = \"\"\n",
    "    analysis_schema = \"\"\n",
    "    job = ModelDeploymentMonitoringJob(\n",
    "        display_name=JOB_NAME,\n",
    "        endpoint=endpoint,\n",
    "        model_deployment_monitoring_objective_configs=objective_configs,\n",
    "        logging_sampling_strategy=sampling_config,\n",
    "        model_deployment_monitoring_schedule_config=schedule_config,\n",
    "        model_monitoring_alert_config=alerting_config,\n",
    "        predict_instance_schema_uri=predict_schema,\n",
    "        analysis_instance_schema_uri=analysis_schema,\n",
    "    )\n",
    "    options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = JobServiceClient(client_options=options)\n",
    "    parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n",
    "    response = client.create_model_deployment_monitoring_job(\n",
    "        parent=parent, model_deployment_monitoring_job=job\n",
    "    )\n",
    "    print(\"Created monitoring job:\")\n",
    "    print(response)\n",
    "    return response\n",
    "\n",
    "\n",
    "def get_thresholds(default_thresholds, custom_thresholds):\n",
    "    thresholds = {}\n",
    "    default_threshold = ThresholdConfig(value=DEFAULT_THRESHOLD_VALUE)\n",
    "    for feature in default_thresholds.split(\",\"):\n",
    "        feature = feature.strip()\n",
    "        thresholds[feature] = default_threshold\n",
    "    for custom_threshold in custom_thresholds.split(\",\"):\n",
    "        pair = custom_threshold.split(\":\")\n",
    "        if len(pair) != 2:\n",
    "            print(f\"Invalid custom skew threshold: {custom_threshold}\")\n",
    "            return\n",
    "        feature, value = pair\n",
    "        thresholds[feature] = ThresholdConfig(value=float(value))\n",
    "    return thresholds\n",
    "\n",
    "\n",
    "def get_deployed_model_ids(endpoint_id):\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = EndpointServiceClient(client_options=client_options)\n",
    "    parent = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n",
    "    response = client.get_endpoint(name=f\"{parent}/endpoints/{endpoint_id}\")\n",
    "    model_ids = []\n",
    "    for model in response.deployed_models:\n",
    "        model_ids.append(model.id)\n",
    "    return model_ids\n",
    "\n",
    "\n",
    "def set_objectives(model_ids, objective_template):\n",
    "    # Use the same objective config for all models.\n",
    "    objective_configs = []\n",
    "    for model_id in model_ids:\n",
    "        objective_config = copy.deepcopy(objective_template)\n",
    "        objective_config.deployed_model_id = model_id\n",
    "        objective_configs.append(objective_config)\n",
    "    return objective_configs\n",
    "\n",
    "\n",
    "def send_predict_request(endpoint, input):\n",
    "    client_options = {\"api_endpoint\": PREDICT_API_ENDPOINT}\n",
    "    client = PredictionServiceClient(client_options=client_options)\n",
    "    params = {}\n",
    "    params = json_format.ParseDict(params, Value())\n",
    "    request = PredictRequest(endpoint=endpoint, parameters=params)\n",
    "    inputs = [json_format.ParseDict(input, Value())]\n",
    "    request.instances.extend(inputs)\n",
    "    response = client.predict(request)\n",
    "    return response\n",
    "\n",
    "\n",
    "def list_monitoring_jobs():\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    parent = f\"projects/{PROJECT_ID}/locations/us-central1\"\n",
    "    client = JobServiceClient(client_options=client_options)\n",
    "    response = client.list_model_deployment_monitoring_jobs(parent=parent)\n",
    "    print(response)\n",
    "\n",
    "\n",
    "def pause_monitoring_job(job):\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = JobServiceClient(client_options=client_options)\n",
    "    response = client.pause_model_deployment_monitoring_job(name=job)\n",
    "    print(response)\n",
    "\n",
    "\n",
    "def delete_monitoring_job(job):\n",
    "    client_options = dict(api_endpoint=API_ENDPOINT)\n",
    "    client = JobServiceClient(client_options=client_options)\n",
    "    response = client.delete_model_deployment_monitoring_job(name=job)\n",
    "    print(response)\n",
    "\n",
    "\n",
    "# Sampling distributions for categorical features...\n",
    "DAYOFWEEK = {1: 1040, 2: 1223, 3: 1352, 4: 1217, 5: 1078, 6: 1011, 7: 1110}\n",
    "LANGUAGE = {\n",
    "    \"en-us\": 4807,\n",
    "    \"en-gb\": 678,\n",
    "    \"ja-jp\": 419,\n",
    "    \"en-au\": 310,\n",
    "    \"en-ca\": 299,\n",
    "    \"de-de\": 147,\n",
    "    \"en-in\": 130,\n",
    "    \"en\": 127,\n",
    "    \"fr-fr\": 94,\n",
    "    \"pt-br\": 81,\n",
    "    \"es-us\": 65,\n",
    "    \"zh-tw\": 64,\n",
    "    \"zh-hans-cn\": 55,\n",
    "    \"es-mx\": 53,\n",
    "    \"nl-nl\": 37,\n",
    "    \"fr-ca\": 34,\n",
    "    \"en-za\": 29,\n",
    "    \"vi-vn\": 29,\n",
    "    \"en-nz\": 29,\n",
    "    \"es-es\": 25,\n",
    "}\n",
    "OS = {\"IOS\": 3980, \"ANDROID\": 3798, \"null\": 253}\n",
    "MONTH = {6: 3125, 7: 1838, 8: 1276, 9: 1718, 10: 74}\n",
    "COUNTRY = {\n",
    "    \"United States\": 4395,\n",
    "    \"India\": 486,\n",
    "    \"Japan\": 450,\n",
    "    \"Canada\": 354,\n",
    "    \"Australia\": 327,\n",
    "    \"United Kingdom\": 303,\n",
    "    \"Germany\": 144,\n",
    "    \"Mexico\": 102,\n",
    "    \"France\": 97,\n",
    "    \"Brazil\": 93,\n",
    "    \"Taiwan\": 72,\n",
    "    \"China\": 65,\n",
    "    \"Saudi Arabia\": 49,\n",
    "    \"Pakistan\": 48,\n",
    "    \"Egypt\": 46,\n",
    "    \"Netherlands\": 45,\n",
    "    \"Vietnam\": 42,\n",
    "    \"Philippines\": 39,\n",
    "    \"South Africa\": 38,\n",
    "}\n",
    "\n",
    "# Means and standard deviations for numerical features...\n",
    "MEAN_SD = {\n",
    "    \"julianday\": (204.6, 34.7),\n",
    "    \"cnt_user_engagement\": (30.8, 53.2),\n",
    "    \"cnt_level_start_quickplay\": (7.8, 28.9),\n",
    "    \"cnt_level_end_quickplay\": (5.0, 16.4),\n",
    "    \"cnt_level_complete_quickplay\": (2.1, 9.9),\n",
    "    \"cnt_level_reset_quickplay\": (2.0, 19.6),\n",
    "    \"cnt_post_score\": (4.9, 13.8),\n",
    "    \"cnt_spend_virtual_currency\": (0.4, 1.8),\n",
    "    \"cnt_ad_reward\": (0.1, 0.6),\n",
    "    \"cnt_challenge_a_friend\": (0.0, 0.3),\n",
    "    \"cnt_completed_5_levels\": (0.1, 0.4),\n",
    "    \"cnt_use_extra_steps\": (0.4, 1.7),\n",
    "}\n",
    "\n",
    "DEFAULT_INPUT = {\n",
    "    \"cnt_ad_reward\": 0,\n",
    "    \"cnt_challenge_a_friend\": 0,\n",
    "    \"cnt_completed_5_levels\": 1,\n",
    "    \"cnt_level_complete_quickplay\": 3,\n",
    "    \"cnt_level_end_quickplay\": 5,\n",
    "    \"cnt_level_reset_quickplay\": 2,\n",
    "    \"cnt_level_start_quickplay\": 6,\n",
    "    \"cnt_post_score\": 34,\n",
    "    \"cnt_spend_virtual_currency\": 0,\n",
    "    \"cnt_use_extra_steps\": 0,\n",
    "    \"cnt_user_engagement\": 120,\n",
    "    \"country\": \"Denmark\",\n",
    "    \"dayofweek\": 3,\n",
    "    \"julianday\": 254,\n",
    "    \"language\": \"da-dk\",\n",
    "    \"month\": 9,\n",
    "    \"operating_system\": \"IOS\",\n",
    "    \"user_pseudo_id\": \"104B0770BAE16E8B53DF330C95881893\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lAOk8UqvCL0S"
   },
   "source": [
    "## Import your model\n",
    "\n",
    "The churn propensity model you'll be using in this notebook has been trained in BigQuery ML and exported to a Google Cloud Storage bucket. This illustrates how you can easily export a trained model and move a model from one cloud service to another. \n",
    "\n",
    "Run the next cell to import this model into your project. **If you've already imported your model, you can skip this step.**\n",
    "\n",
    "**NOTE: If you encounter a model building error, for example, \"service account doesn't have sufficient IAM permission(s) for writing objects to the Google Cloud Storage bucket\" , try to re-run the cell.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "Se5DxJoK3AMz"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model output:  ['Using endpoint [https://us-central1-aiplatform.googleapis.com/]', 'projects/346810986095/locations/us-central1/models/4351946187574607872', 'Waiting for operation [4975491601192189952]...', '................................................................................................done.']\n",
      "Model churn/4351946187574607872 created.\n"
     ]
    }
   ],
   "source": [
    "# TODO\n",
    "# Import your model\n",
    "MODEL_NAME = \"churn\"\n",
    "IMAGE = \"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-6:latest\"\n",
    "ARTIFACT = \"gs://mco-mm/churn\"\n",
    "output = !gcloud --quiet ai models upload --container-image-uri=$IMAGE --artifact-uri=$ARTIFACT --display-name=$MODEL_NAME --format=\"value(model)\"\n",
    "print(\"model output: \", output)\n",
    "MODEL_ID = output[-1].split('/')[-1]\n",
    "print(f\"Model {MODEL_NAME}/{MODEL_ID} created.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L1HzDTh-JRsr"
   },
   "source": [
    "## Deploy your endpoint\n",
    "\n",
    "Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you'll be able to make prediction requests on your model via the public internet.  Your endpoint is also serverless, in the sense that Google ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.\n",
    "\n",
    "Run the next cell to deploy your model to an endpoint. **This will take about ten minutes to complete. If you've already deployed a model to an endpoint, you can reuse your endpoint by running the cell after the next one.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "M4nHQTVh7PIi"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "endpoint output:  ['Using endpoint [https://us-central1-aiplatform.googleapis.com/]', 'Waiting for operation [5397704066258173952]...', '........................................done.', 'Created Vertex AI endpoint: projects/346810986095/locations/us-central1/endpoints/3623550520498585600.', 'projects/346810986095/locations/us-central1/endpoints/3623550520498585600']\n",
      "Model churn/4351946187574607872/[4171599067706556416].. deployed to Endpoint churn/3623550520498585600/projects/346810986095/locations/us-central1/endpoints/3623550520498585600.\n"
     ]
    }
   ],
   "source": [
    "# TODO\n",
    "# Deploy your model to the endpoint\n",
    "ENDPOINT_NAME = \"churn\"\n",
    "output = !gcloud --quiet ai endpoints create --display-name=$ENDPOINT_NAME --format=\"value(name)\"\n",
    "print(\"endpoint output: \", output)\n",
    "ENDPOINT = output[-1]\n",
    "ENDPOINT_ID = ENDPOINT.split(\"/\")[-1]\n",
    "output = !gcloud --quiet ai endpoints deploy-model $ENDPOINT_ID --display-name=$ENDPOINT_NAME --model=$MODEL_ID --traffic-split=\"0=100\"\n",
    "DEPLOYED_MODEL_ID = output[1].split()[-1][:-1]\n",
    "print(\n",
    "    f\"Model {MODEL_NAME}/{MODEL_ID}/{DEPLOYED_MODEL_ID} deployed to Endpoint {ENDPOINT_NAME}/{ENDPOINT_ID}/{ENDPOINT}.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NKsA_lfl9Ryw"
   },
   "source": [
    "## Run a prediction test\n",
    "\n",
    "Now that you have imported a model and deployed that model to an endpoint, you are ready to verify that it's working. Run the next cell to send a test prediction request. If everything works as expected, you should receive a response encoded in a text representation called JSON.\n",
    "\n",
    "**Try this now by running the next cell and examine the results.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "QNEb7fDJ9NXc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/346810986095/locations/us-central1/endpoints/3623550520498585600\n",
      "request:\n",
      "{'cnt_ad_reward': 0,\n",
      " 'cnt_challenge_a_friend': 0,\n",
      " 'cnt_completed_5_levels': 1,\n",
      " 'cnt_level_complete_quickplay': 3,\n",
      " 'cnt_level_end_quickplay': 5,\n",
      " 'cnt_level_reset_quickplay': 2,\n",
      " 'cnt_level_start_quickplay': 6,\n",
      " 'cnt_post_score': 34,\n",
      " 'cnt_spend_virtual_currency': 0,\n",
      " 'cnt_use_extra_steps': 0,\n",
      " 'cnt_user_engagement': 120,\n",
      " 'country': 'Denmark',\n",
      " 'dayofweek': 3,\n",
      " 'julianday': 254,\n",
      " 'language': 'da-dk',\n",
      " 'month': 9,\n",
      " 'operating_system': 'IOS',\n",
      " 'user_pseudo_id': '104B0770BAE16E8B53DF330C95881893'}\n",
      "response\n",
      "predictions {\n",
      "  struct_value {\n",
      "    fields {\n",
      "      key: \"churned_probs\"\n",
      "      value {\n",
      "        list_value {\n",
      "          values {\n",
      "            number_value: 0.8705776921658448\n",
      "          }\n",
      "          values {\n",
      "            number_value: 0.1294223078341552\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "    fields {\n",
      "      key: \"churned_values\"\n",
      "      value {\n",
      "        list_value {\n",
      "          values {\n",
      "            string_value: \"1\"\n",
      "          }\n",
      "          values {\n",
      "            string_value: \"0\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "    fields {\n",
      "      key: \"predicted_churned\"\n",
      "      value {\n",
      "        list_value {\n",
      "          values {\n",
      "            string_value: \"1\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "deployed_model_id: \"2157655230068555776\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pprint as pp\n",
    "\n",
    "print(ENDPOINT)\n",
    "print(\"request:\")\n",
    "pp.pprint(DEFAULT_INPUT)\n",
    "try:\n",
    "    resp = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n",
    "    print(\"response\")\n",
    "    pp.pprint(resp)\n",
    "except Exception:\n",
    "    print(\"prediction request failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lN6ADMJqeWtR"
   },
   "source": [
    "Taking a closer look at the results, we see the following elements:\n",
    "\n",
    "- **churned_values** - a set of possible values (0 and 1) for the target field\n",
    "- **churned_probs** - a corresponding set of probabilities for each possible target field value (5x10^-40 and 1.0, respectively)\n",
    "- **predicted_churn** - based on the probabilities, the predicted value of the target field (1)\n",
    "\n",
    "This response encodes the model's prediction in a format that is readily digestible by software, which makes this service ideal for automated use by an application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8rF5iLuXCT7i"
   },
   "source": [
    "## Start your monitoring job\n",
    "\n",
    "Now that you've created an endpoint to serve prediction requests on your model, you're ready to start a monitoring job to keep an eye on model quality and to alert you if and when input begins to deviate in way that may impact your model's prediction quality.\n",
    "\n",
    "In this section, you will configure and create a model monitoring job based on the churn propensity model you imported from BigQuery ML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wW2gLBQ3Zkhq"
   },
   "source": [
    "### Configure the following fields:\n",
    "\n",
    "1. Log sample rate - Your prediction requests and responses are logged to BigQuery tables, which are automatically created when you create a monitoring job. This parameter specifies the desired logging frequency for those tables.\n",
    "1. Monitor interval - the  time window over which to analyze your data and report anomalies. The minimum window is one hour (3600 seconds).\n",
    "1. Target field - the prediction target column name in training dataset.\n",
    "1. Skew detection threshold - the skew threshold for each feature you want to monitor.\n",
    "1. Prediction drift threshold - the drift threshold for each feature you want to monitor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "plpASmM2YIVO"
   },
   "outputs": [],
   "source": [
    "USER_EMAIL = \"[insert_your_personal_email_address_to_receive_job_notifications]\" \n",
    "JOB_NAME = \"churn\"\n",
    "\n",
    "# Sampling rate (optional, default=.8)\n",
    "LOG_SAMPLE_RATE = 0.8 \n",
    "\n",
    "# Monitoring Interval in seconds (optional, default=3600).\n",
    "MONITOR_INTERVAL = 3600 \n",
    "\n",
    "# URI to training dataset.\n",
    "DATASET_BQ_URI = \"bq://mco-mm.bqmlga4.train\" \n",
    "# Prediction target column name in training dataset.\n",
    "TARGET = \"churned\"\n",
    "\n",
    "# Skew and drift thresholds.\n",
    "SKEW_DEFAULT_THRESHOLDS = \"country,language\" \n",
    "SKEW_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\" \n",
    "DRIFT_DEFAULT_THRESHOLDS = \"country,language\" \n",
    "DRIFT_CUSTOM_THRESHOLDS = \"cnt_user_engagement:.5\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mjVSViZR-dP2"
   },
   "source": [
    "### Create your monitoring job\n",
    "\n",
    "The following code uses the Google Python client library to translate your configuration settings into a programmatic request to start a model monitoring job. Instantiating a monitoring job can take some time. If everything looks good with your request, you'll get a successful API response. Then, you'll need to check your email to receive a notification that the job is running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "-62TYm2iYv3K"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created monitoring job:\n",
      "name: \"projects/346810986095/locations/us-central1/modelDeploymentMonitoringJobs/7559047882960011264\"\n",
      "display_name: \"churn\"\n",
      "endpoint: \"projects/346810986095/locations/us-central1/endpoints/3623550520498585600\"\n",
      "state: JOB_STATE_PENDING\n",
      "schedule_state: OFFLINE\n",
      "model_deployment_monitoring_objective_configs {\n",
      "  deployed_model_id: \"2157655230068555776\"\n",
      "  objective_config {\n",
      "    training_dataset {\n",
      "      bigquery_source {\n",
      "        input_uri: \"bq://mco-mm.bqmlga4.train\"\n",
      "      }\n",
      "      target_field: \"churned\"\n",
      "    }\n",
      "    training_prediction_skew_detection_config {\n",
      "      skew_thresholds {\n",
      "        key: \"cnt_user_engagement\"\n",
      "        value {\n",
      "          value: 0.5\n",
      "        }\n",
      "      }\n",
      "      skew_thresholds {\n",
      "        key: \"country\"\n",
      "        value {\n",
      "          value: 0.001\n",
      "        }\n",
      "      }\n",
      "      skew_thresholds {\n",
      "        key: \"language\"\n",
      "        value {\n",
      "          value: 0.001\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "    prediction_drift_detection_config {\n",
      "      drift_thresholds {\n",
      "        key: \"cnt_user_engagement\"\n",
      "        value {\n",
      "          value: 0.5\n",
      "        }\n",
      "      }\n",
      "      drift_thresholds {\n",
      "        key: \"country\"\n",
      "        value {\n",
      "          value: 0.001\n",
      "        }\n",
      "      }\n",
      "      drift_thresholds {\n",
      "        key: \"language\"\n",
      "        value {\n",
      "          value: 0.001\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "model_deployment_monitoring_schedule_config {\n",
      "  monitor_interval {\n",
      "    seconds: 3600\n",
      "  }\n",
      "}\n",
      "logging_sampling_strategy {\n",
      "  random_sample_config {\n",
      "    sample_rate: 0.8\n",
      "  }\n",
      "}\n",
      "bigquery_tables {\n",
      "  log_source: SERVING\n",
      "  log_type: PREDICT\n",
      "  bigquery_table_path: \"bq://qwiklabs-gcp-04-9946b386a017.model_deployment_monitoring_3623550520498585600.serving_predict\"\n",
      "}\n",
      "create_time {\n",
      "  seconds: 1630149944\n",
      "  nanos: 75605000\n",
      "}\n",
      "update_time {\n",
      "  seconds: 1630149944\n",
      "  nanos: 75605000\n",
      "}\n",
      "next_schedule_time {\n",
      "}\n",
      "model_monitoring_alert_config {\n",
      "  email_alert_config {\n",
      "    user_emails: \" \"\n",
      "  }\n",
      "}\n",
      "log_ttl {\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# TODO\n",
    "# Create a monitoring job\n",
    "skew_thresholds = get_thresholds(SKEW_DEFAULT_THRESHOLDS, SKEW_CUSTOM_THRESHOLDS)\n",
    "drift_thresholds = get_thresholds(DRIFT_DEFAULT_THRESHOLDS, DRIFT_CUSTOM_THRESHOLDS)\n",
    "skew_config = ModelMonitoringObjectiveConfig.TrainingPredictionSkewDetectionConfig(\n",
    "    skew_thresholds=skew_thresholds\n",
    ")\n",
    "drift_config = ModelMonitoringObjectiveConfig.PredictionDriftDetectionConfig(\n",
    "    drift_thresholds=drift_thresholds\n",
    ")\n",
    "training_dataset = ModelMonitoringObjectiveConfig.TrainingDataset(target_field=TARGET)\n",
    "training_dataset.bigquery_source = BigQuerySource(input_uri=DATASET_BQ_URI)\n",
    "objective_config = ModelMonitoringObjectiveConfig(\n",
    "    training_dataset=training_dataset,\n",
    "    training_prediction_skew_detection_config=skew_config,\n",
    "    prediction_drift_detection_config=drift_config,\n",
    ")\n",
    "model_ids = get_deployed_model_ids(ENDPOINT_ID)\n",
    "objective_template = ModelDeploymentMonitoringObjectiveConfig(\n",
    "    objective_config=objective_config\n",
    ")\n",
    "objective_configs = set_objectives(model_ids, objective_template)\n",
    "\n",
    "monitoring_job = create_monitoring_job(objective_configs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "OiwOVR4D_xhl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prediction succeeded\n"
     ]
    }
   ],
   "source": [
    "# Run a prediction request to generate schema, if necessary.\n",
    "try:\n",
    "    _ = send_predict_request(ENDPOINT, DEFAULT_INPUT)\n",
    "    print(\"prediction succeeded\")\n",
    "except Exception:\n",
    "    print(\"prediction failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SaXYVFFslRru"
   },
   "source": [
    "After a minute or two, you should receive email at the address you configured above for USER_EMAIL. This email confirms successful deployment of your monitoring job. Here's a sample of what this email might look like:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![monitoring_job.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SaXYVFFslRru"
   },
   "source": [
    "As your monitoring job collects data, measurements are stored in Google Cloud Storage and you are free to examine your data at any time. The circled path in the image above specifies the location of your measurements in Google Cloud Storage. Run the following cell to take a look at your measurements in Cloud Storage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "XV-vru2Pm1oX"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/instance_schemas/job-7195547139794010112/:\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/instance_schemas/job-7195547139794010112/analysis\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/instance_schemas/job-7195547139794010112/predict\n",
      "\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/instance_schemas/job-8062415301272141824/:\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/instance_schemas/job-8062415301272141824/analysis\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/instance_schemas/job-8062415301272141824/predict\n",
      "\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/model_monitoring/job-7195547139794010112/:\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/model_monitoring/job-7195547139794010112/serving/\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/model_monitoring/job-7195547139794010112/training/\n",
      "\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/model_monitoring/job-8062415301272141824/:\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/model_monitoring/job-8062415301272141824/serving/\n",
      "gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/model_monitoring/job-8062415301272141824/training/\n"
     ]
    }
   ],
   "source": [
    "!gcloud storage ls gs://cloud-ai-platform-fdfb4810-148b-4c86-903c-dbdff879f6e1/*/*"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XgUwU0sDpUUD"
   },
   "source": [
    "You will notice the following components in these Cloud Storage paths:\n",
    "\n",
    "- **cloud-ai-platform-..** - This is a bucket created for you and assigned to capture your service's prediction data. Each monitoring job you create will trigger creation of a new folder in this bucket.\n",
    "- **[model_monitoring|instance_schemas]/job-..** - This is your unique monitoring job number, which you can see above in both the response to your job creation requesst and the email notification. \n",
    "- **instance_schemas/job-../analysis** - This is the monitoring jobs understanding and encoding of your training data's schema (field names, types, etc.).\n",
    "- **instance_schemas/job-../predict** - This is the first prediction made to your model after the current monitoring job was enabled.\n",
    "- **model_monitoring/job-../serving** - This folder is used to record data relevant to drift calculations. It contains measurement summaries for every hour your model serves traffic.\n",
    "- **model_monitoring/job-../training** - This folder is used to record data relevant to training-serving skew calculations. It contains an ongoing summary of prediction data relative to training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8V2zo7-MMd7G"
   },
   "source": [
    "### You can create monitoring jobs with other user interfaces\n",
    "\n",
    "In the previous cells, you created a monitoring job using the Python client library. You can also use the *gcloud* command line tool to create a model monitoring job and, in the near future, you will be able to use the Cloud Console, as well for this function. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q106INuFCXKX"
   },
   "source": [
    "## Generate test data to trigger alerting\n",
    "\n",
    "Now you are ready to test the monitoring function. Run the following cell, which will generate fabricated test predictions designed to trigger the thresholds you specified above. It takes about five minutes to run this cell and at least an hour to assess and report anamolies in skew or drift so after running this cell, feel free to proceed with the notebook and you'll see how to examine the resulting alert later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "obZYLLAuKmG8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending prediction 0\n",
      "Sending prediction 1\n",
      "Sending prediction 2\n",
      "Sending prediction 3\n",
      "Sending prediction 4\n",
      "Sending prediction 5\n",
      "Sending prediction 6\n",
      "Sending prediction 7\n",
      "Sending prediction 8\n",
      "Sending prediction 9\n",
      "Sending prediction 10\n",
      "Sending prediction 11\n",
      "Sending prediction 12\n",
      "Sending prediction 13\n",
      "Sending prediction 14\n",
      "Sending prediction 15\n",
      "Sending prediction 16\n",
      "Sending prediction 17\n",
      "Sending prediction 18\n",
      "Sending prediction 19\n",
      "Sending prediction 20\n",
      "Sending prediction 21\n",
      "Sending prediction 22\n",
      "Sending prediction 23\n",
      "Sending prediction 24\n",
      "Sending prediction 25\n",
      "Sending prediction 26\n",
      "Sending prediction 27\n",
      "Sending prediction 28\n",
      "Sending prediction 29\n",
      "Sending prediction 30\n",
      "Sending prediction 31\n",
      "Sending prediction 32\n",
      "Sending prediction 33\n",
      "Sending prediction 34\n",
      "Sending prediction 35\n",
      "Sending prediction 36\n",
      "Sending prediction 37\n",
      "Sending prediction 38\n",
      "Sending prediction 39\n",
      "Sending prediction 40\n",
      "Sending prediction 41\n",
      "Sending prediction 42\n",
      "Sending prediction 43\n",
      "Sending prediction 44\n",
      "Sending prediction 45\n",
      "Sending prediction 46\n",
      "Sending prediction 47\n",
      "Sending prediction 48\n",
      "Sending prediction 49\n",
      "Sending prediction 50\n",
      "Sending prediction 51\n",
      "Sending prediction 52\n",
      "Sending prediction 53\n",
      "Sending prediction 54\n",
      "Sending prediction 55\n",
      "Sending prediction 56\n",
      "Sending prediction 57\n",
      "Sending prediction 58\n",
      "Sending prediction 59\n",
      "Sending prediction 60\n",
      "Sending prediction 61\n",
      "Sending prediction 62\n",
      "Sending prediction 63\n",
      "Sending prediction 64\n",
      "Sending prediction 65\n",
      "Sending prediction 66\n",
      "Sending prediction 67\n",
      "Sending prediction 68\n",
      "Sending prediction 69\n",
      "Sending prediction 70\n",
      "Sending prediction 71\n",
      "Sending prediction 72\n",
      "Sending prediction 73\n",
      "Sending prediction 74\n",
      "Sending prediction 75\n",
      "Sending prediction 76\n",
      "Sending prediction 77\n",
      "Sending prediction 78\n",
      "Sending prediction 79\n",
      "Sending prediction 80\n",
      "Sending prediction 81\n",
      "Sending prediction 82\n",
      "Sending prediction 83\n",
      "Sending prediction 84\n",
      "Sending prediction 85\n",
      "Sending prediction 86\n",
      "Sending prediction 87\n",
      "Sending prediction 88\n",
      "Sending prediction 89\n",
      "Sending prediction 90\n",
      "Sending prediction 91\n",
      "Sending prediction 92\n",
      "Sending prediction 93\n",
      "Sending prediction 94\n",
      "Sending prediction 95\n",
      "Sending prediction 96\n",
      "Sending prediction 97\n",
      "Sending prediction 98\n",
      "Sending prediction 99\n",
      "Sending prediction 100\n",
      "Sending prediction 101\n",
      "Sending prediction 102\n",
      "Sending prediction 103\n",
      "Sending prediction 104\n",
      "Sending prediction 105\n",
      "Sending prediction 106\n",
      "Sending prediction 107\n",
      "Sending prediction 108\n",
      "Sending prediction 109\n",
      "Sending prediction 110\n",
      "Sending prediction 111\n",
      "Sending prediction 112\n",
      "Sending prediction 113\n",
      "Sending prediction 114\n",
      "Sending prediction 115\n",
      "Sending prediction 116\n",
      "Sending prediction 117\n",
      "Sending prediction 118\n",
      "Sending prediction 119\n",
      "Sending prediction 120\n",
      "Sending prediction 121\n",
      "Sending prediction 122\n",
      "Sending prediction 123\n",
      "Sending prediction 124\n",
      "Sending prediction 125\n",
      "Sending prediction 126\n",
      "Sending prediction 127\n",
      "Sending prediction 128\n",
      "Sending prediction 129\n",
      "Sending prediction 130\n",
      "Sending prediction 131\n",
      "Sending prediction 132\n",
      "Sending prediction 133\n",
      "Sending prediction 134\n",
      "Sending prediction 135\n",
      "Sending prediction 136\n",
      "Sending prediction 137\n",
      "Sending prediction 138\n",
      "Sending prediction 139\n",
      "Sending prediction 140\n",
      "Sending prediction 141\n",
      "Sending prediction 142\n",
      "Sending prediction 143\n",
      "Sending prediction 144\n",
      "Sending prediction 145\n",
      "Sending prediction 146\n",
      "Sending prediction 147\n",
      "Sending prediction 148\n",
      "Sending prediction 149\n",
      "Sending prediction 150\n",
      "Sending prediction 151\n",
      "Sending prediction 152\n",
      "Sending prediction 153\n",
      "Sending prediction 154\n",
      "Sending prediction 155\n",
      "Sending prediction 156\n",
      "Sending prediction 157\n",
      "Sending prediction 158\n",
      "Sending prediction 159\n",
      "Sending prediction 160\n",
      "Sending prediction 161\n",
      "Sending prediction 162\n",
      "Sending prediction 163\n",
      "Sending prediction 164\n",
      "Sending prediction 165\n",
      "Sending prediction 166\n",
      "Sending prediction 167\n",
      "Sending prediction 168\n",
      "Sending prediction 169\n",
      "Sending prediction 170\n",
      "Sending prediction 171\n",
      "Sending prediction 172\n",
      "Sending prediction 173\n",
      "Sending prediction 174\n",
      "Sending prediction 175\n",
      "Sending prediction 176\n",
      "Sending prediction 177\n",
      "Sending prediction 178\n",
      "Sending prediction 179\n",
      "Sending prediction 180\n",
      "Sending prediction 181\n",
      "Sending prediction 182\n",
      "Sending prediction 183\n",
      "Sending prediction 184\n",
      "Sending prediction 185\n",
      "Sending prediction 186\n",
      "Sending prediction 187\n",
      "Sending prediction 188\n",
      "Sending prediction 189\n",
      "Sending prediction 190\n",
      "Sending prediction 191\n",
      "Sending prediction 192\n",
      "Sending prediction 193\n",
      "Sending prediction 194\n",
      "Sending prediction 195\n",
      "Sending prediction 196\n",
      "Sending prediction 197\n",
      "Sending prediction 198\n",
      "Sending prediction 199\n",
      "Sending prediction 200\n",
      "Sending prediction 201\n",
      "Sending prediction 202\n",
      "Sending prediction 203\n",
      "Sending prediction 204\n",
      "Sending prediction 205\n",
      "Sending prediction 206\n",
      "Sending prediction 207\n",
      "Sending prediction 208\n",
      "Sending prediction 209\n",
      "Sending prediction 210\n",
      "Sending prediction 211\n",
      "Sending prediction 212\n",
      "Sending prediction 213\n",
      "Sending prediction 214\n",
      "Sending prediction 215\n",
      "Sending prediction 216\n",
      "Sending prediction 217\n",
      "Sending prediction 218\n",
      "Sending prediction 219\n",
      "Sending prediction 220\n",
      "Sending prediction 221\n",
      "Sending prediction 222\n",
      "Sending prediction 223\n",
      "Sending prediction 224\n",
      "Sending prediction 225\n",
      "Sending prediction 226\n",
      "Sending prediction 227\n",
      "Sending prediction 228\n",
      "Sending prediction 229\n",
      "Sending prediction 230\n",
      "Sending prediction 231\n",
      "Sending prediction 232\n",
      "Sending prediction 233\n",
      "Sending prediction 234\n",
      "Sending prediction 235\n",
      "Sending prediction 236\n",
      "Sending prediction 237\n",
      "Sending prediction 238\n",
      "Sending prediction 239\n",
      "Sending prediction 240\n",
      "Sending prediction 241\n",
      "Sending prediction 242\n",
      "Sending prediction 243\n",
      "Sending prediction 244\n",
      "Sending prediction 245\n",
      "Sending prediction 246\n",
      "Sending prediction 247\n",
      "Sending prediction 248\n",
      "Sending prediction 249\n",
      "Sending prediction 250\n",
      "Sending prediction 251\n",
      "Sending prediction 252\n",
      "Sending prediction 253\n",
      "Sending prediction 254\n",
      "Sending prediction 255\n",
      "Sending prediction 256\n",
      "Sending prediction 257\n",
      "Sending prediction 258\n",
      "Sending prediction 259\n",
      "Sending prediction 260\n",
      "Sending prediction 261\n",
      "Sending prediction 262\n",
      "Sending prediction 263\n",
      "Sending prediction 264\n",
      "Sending prediction 265\n",
      "Sending prediction 266\n",
      "Sending prediction 267\n",
      "Sending prediction 268\n",
      "Sending prediction 269\n",
      "Sending prediction 270\n",
      "Sending prediction 271\n",
      "Sending prediction 272\n",
      "Sending prediction 273\n",
      "Sending prediction 274\n",
      "Sending prediction 275\n",
      "Sending prediction 276\n",
      "Sending prediction 277\n",
      "Sending prediction 278\n",
      "Sending prediction 279\n",
      "Sending prediction 280\n",
      "Sending prediction 281\n",
      "Sending prediction 282\n",
      "Sending prediction 283\n",
      "Sending prediction 284\n",
      "Sending prediction 285\n",
      "Sending prediction 286\n",
      "Sending prediction 287\n",
      "Sending prediction 288\n",
      "Sending prediction 289\n",
      "Sending prediction 290\n",
      "Sending prediction 291\n",
      "Sending prediction 292\n",
      "Sending prediction 293\n",
      "Sending prediction 294\n",
      "Sending prediction 295\n",
      "Sending prediction 296\n",
      "Sending prediction 297\n",
      "Sending prediction 298\n",
      "Sending prediction 299\n",
      "Test Completed.\n"
     ]
    }
   ],
   "source": [
    "# TODO\n",
    "# Generate test predictions to trigger the thresholds\n",
    "def random_uid():\n",
    "    digits = [str(i) for i in range(10)] + [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"]\n",
    "    return \"\".join(random.choices(digits, k=32))\n",
    "\n",
    "\n",
    "def monitoring_test(count, sleep, perturb_num={}, perturb_cat={}):\n",
    "    # Use random sampling and mean/sd with gaussian distribution to model\n",
    "    # training data. Then modify sampling distros for two categorical features\n",
    "    # and mean/sd for two numerical features.\n",
    "    mean_sd = MEAN_SD.copy()\n",
    "    country = COUNTRY.copy()\n",
    "    for k, (mean_fn, sd_fn) in perturb_num.items():\n",
    "        orig_mean, orig_sd = MEAN_SD[k]\n",
    "        mean_sd[k] = (mean_fn(orig_mean), sd_fn(orig_sd))\n",
    "    for k, v in perturb_cat.items():\n",
    "        country[k] = v\n",
    "    for i in range(0, count):\n",
    "        input = DEFAULT_INPUT.copy()\n",
    "        input[\"user_pseudo_id\"] = str(random_uid())\n",
    "        input[\"country\"] = random.choices([*country], list(country.values()))[0]\n",
    "        input[\"dayofweek\"] = random.choices([*DAYOFWEEK], list(DAYOFWEEK.values()))[0]\n",
    "        input[\"language\"] = str(random.choices([*LANGUAGE], list(LANGUAGE.values()))[0])\n",
    "        input[\"operating_system\"] = str(random.choices([*OS], list(OS.values()))[0])\n",
    "        input[\"month\"] = random.choices([*MONTH], list(MONTH.values()))[0]\n",
    "        for key, (mean, sd) in mean_sd.items():\n",
    "            sample_val = round(float(np.random.normal(mean, sd, 1)))\n",
    "            val = max(sample_val, 0)\n",
    "            input[key] = val\n",
    "        print(f\"Sending prediction {i}\")\n",
    "        try:\n",
    "            send_predict_request(ENDPOINT, input)\n",
    "        except Exception:\n",
    "            print(\"prediction request failed\")\n",
    "        time.sleep(sleep)\n",
    "    print(\"Test Completed.\")\n",
    "\n",
    "\n",
    "test_time = 300\n",
    "tests_per_sec = 1\n",
    "sleep_time = 1 / tests_per_sec\n",
    "iterations = test_time * tests_per_sec\n",
    "perturb_num = {\"cnt_user_engagement\": (lambda x: x * 3, lambda x: x / 3)}\n",
    "perturb_cat = {\"Japan\": max(COUNTRY.values()) * 2}\n",
    "monitoring_test(iterations, sleep_time, perturb_num, perturb_cat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bQohDTJgLQlW"
   },
   "source": [
    "## Interpret your results\n",
    "\n",
    "While waiting for your results, which, as noted, may take up to an hour and half, you can read ahead to get sense of the alerting experience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uGPI92qbOFUR"
   },
   "source": [
    "### Here's what a sample email alert looks like...\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![prediction.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HoaqsxpaRs1m"
   },
   "source": [
    "This email is warning you that the *cnt_user_engagement*, *country* and *language* feature values seen in production have skewed above your threshold between training and serving your model. It's also telling you that the *cnt_user_engagement* feature value is drifting significantly over time, again, as per your threshold specification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w4jVIVq4VzB_"
   },
   "source": [
    "### Monitoring results in the Cloud Console\n",
    "\n",
    "You can examine your model monitoring data from the Cloud Console. Below is a screenshot of those capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2OdIMVBAPZi_"
   },
   "source": [
    "#### Monitoring Status\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alert.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2OdIMVBAPZi_"
   },
   "source": [
    "#### Monitoring Alerts\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alert2.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j3Dh15h3-NoO"
   },
   "source": [
    "## Learn more about model monitoring\n",
    "\n",
    "**Congratulations!** You've now learned what model monitoring is, how to configure and enable it, and how to find and interpret the results. Check out the following resources to learn more about model monitoring and ML Ops.\n",
    "\n",
    "- [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv)\n",
    "- [Data Understanding, Validation, and Monitoring At Scale](https://blog.tensorflow.org/2018/09/introducing-tensorflow-data-validation.html)\n",
    "- [Vertex Product Documentation](https://cloud.google.com/vertex-ai/docs)\n",
    "- [Model Monitoring Reference Docs](https://cloud.google.com/vertex-ai/docs/model-monitoring)\n",
    "- [Model Monitoring blog article](https://cloud.google.com/blog/topics/developers-practitioners/monitor-models-training-serving-skew-vertex-ai)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "lA32H1oKGgpf"
   ],
   "name": "model_monitoring.ipynb",
   "toc_visible": true
  },
  "environment": {
   "name": "tf2-gpu.2-5.m76",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-5:m76"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
